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Analysis  of  Team  Collaboration  for  Two  Decision-Making  Domains 
to  Empirically  Evaluate  a  Model  of  Team  Collaboration 

Abstract 

A  model  of  team  collaboration  was  empirically  evaluated  by  applying  definitions 
of  the  macrocognitive  processes  included  in  the  model  to  the  team  communi¬ 
cations  that  transpired  during  two  real-world  decision-making  domains.  Macroco¬ 
gnition  is  defined  as  the  internalized  and  externalized  high-level  mental  processes 
employed  by  teams  to  create  new  knowledge  during  complex,  one-of-a-kind 
problem  solving.  Team  collaboration  during  two  unique,  information-rich,  time- 
compressed  situations  was  analyzed  and  coded,  using  verbatim  transcripts  or  chat 
logs,  to  empirically  evaluate  the  model  of  team  collaboration.  Many  tasks  require 
rapid  and  accurate  coordination  of  information,  expertise,  perspectives  and 
behavior  to  successfully  cope  with  the  demands  of  time-compressed,  ambiguous 
situations  with  the  threat  of  fatal  consequences.  We  analyzed  team  communi¬ 
cations  from  the  North  American  Aerospace  Defense  Command  collaborating 
with  other  agencies  to  deal  with  the  Sept  11,  2001,  hijacking  of  four  US 
commercial  aircraft  and  an  Air  Operations  Center  team  collaborating  to  conduct 
time-sensitive  targeting.  A  new  macrocognitive  process  emerged  during  the 
coding  process:  decision  to  take  action.  Deciding  to  take  action  is  viewed  as  both 
a  macrocognitive  process  and  a  product  of  team  collaboration.  Results  indicate 
additional  macrocognitive  processes  need  to  be  included  in  the  model  to  represent 
decision  making  which  occurs  during  execution  of  real-world  tasks. 

INTRODUCTION 

One  goal  for  studying  macrocognition  is  to  understand  the  complexity  entailed  in  inter-  and 
intra-individual  cognition.  Macrocognition  is  an  emerging  field  within  the  area  of  cognitive 
engineering  that  describes  the  way  cognition  occurs  in  naturalistic,  or  real-world,  decision¬ 
making  events  (Cacciabue  &  Hollnagell,  1995).  From  this  perspective,  macrocognition 
comprises  the  mental  activities  that  must  be  successfully  accomplished  to  perform  a  task  or 
achieve  a  goal  (Klein,  Ross,  Moon,  Klein,  Hoffman,  &  Hollnagell,  2003).  Macrocognitive 
functions  are  generally  performed  during  collaborative  team  problem  solving,  where  the 
emphasis  is  on  building  new  knowledge. 

For  the  research  reported  in  this  paper,  we  focus  on  cognition  in  collaboration  contexts  (Warner, 
Letsky,  &  Cowen,  2005),  where  problem  solving  teams  collaborate  on  short-term  situations 
which  require  rapid  action  to  be  taken  against  specific  missions  (Letsky,  Warner,  Fiore,  Rosen, 
&  Salas,  2007).  Macrocognition  is  defined  as  the  internalized  and  externalized  high-level  mental 
processes  employed  by  teams  to  create  new  knowledge  during  complex,  one-of-a  kind, 
collaborative  problem  solving  (Letsky  et  ah,  2007).  High-level  mental  processes  refer  to  the 
cognitive  processes  involved  in  combining,  visualizing,  and  aggregating  information  to  resolve 
ambiguity  in  support  of  the  discovery  of  new  knowledge  and  relationships. 

This  conceptualization  of  macrocognition  views  it  as  a  complex,  multi-level  phenomenon  that 
involves  development,  refinement,  and  maintenance  of  higher-order  cognitive  processes  and 
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emergent  states  (Burke,  2007).  Macrocognition  involves  the  cognitive  processes  employed  by 
team  members  in  unique,  information-rich,  time-compressed  collaborative  problem  solving,  such 
as  individual  and  team  knowledge  development,  shared  problem  conceptualization,  mental 
model  development,  and  solution  option  generation.  Macrocognition  encompasses  detecting 
problems,  developing  and  sharing  situation  awareness,  generating  options,  using  analogues, 
mentally  simulating  courses  if  action,  planning  and  re-planning,  maintaining  vigilance,  and 
assessing  risk  (Klein,  2001). 

The  framework  of  collaborative  problem  solving  developed  as  part  of  the  Office  of  Naval 
Research  (ONR)  Collaboration  and  Knowledge  Integration  (CKI)  Program  (Letsky,  Warner, 
Fiore,  &  Salas,  2007)  provides  the  conceptual  foundation  for  this  research.  Problem-solving 
domains  analyzed  previously  for  this  research  include:  firefighters  responding  to  the  events  of 
Sept.  11,  2001,  a  Coast  Guard  boarding  party  conducting  Maritime  Interdiction  Operations 
(MIO)  on  a  ship  suspected  of  carrying  contraband  material,  and  air  warfare  teams  on  a  Navy  ship 
(Hutchins  &  Kendall,  in  press;  Hutchins  &  Kendall,  2008).  Information  integration,  or 
knowledge  construction,  was  a  major  contributor  to  performance  for  all  teams  who  were 
responding  to  the  various  problem-solving  situations  we  studied. 

The  goal  for  the  research  reported  here  is  to  understand  the  role  of  cognition  in  teams  who  are 
collaborating  to  solve  challenging,  ambiguous  problems.  Our  objectives  are  to  (1)  empirically 
evaluate  the  model  of  team  collaboration  based  upon  analysis  of  real-world  complex  decision¬ 
making  events  and  (2)  develop  a  better  understanding  of  the  cognitive  processes  employed  when 
teams  collaborate  to  solve  problems. 

Team  Collaboration 

Many  definitions  of  collaboration  are  found  in  the  different  bodies  of  research  literature. 
Collaboration  derives  from  the  Latin  collaboratus,  past  participle  of  collaborare,  to  labor,  and 
one  definition  is  to  work  jointly  with  others  or  together  especially  in  an  intellectual  endeavor. 
Collaboration  provides  increased  information  processing  capacity  where  more  minds  are  enlisted 
to  handle  complex  problems  (Hocevar,  Jansen,  and  Thomas,  2004).  Team  members  provide 
several  perspectives  on  an  issue  for  generating,  choosing,  and  implementing  action  plans.  A 
collaborative  approach  also  provides  greater  flexibility  and  innovation  where  human  judgment 
and  experience  are  leveraged  (Hocevar,  et  al,  2004.)  Collaboration  occurs  “when  a  group  of 
autonomous  stakeholders  of  a  problem  domain  engage  in  an  interactive  process,  using  shared 
rules,  nonns,  and  structures  to  act  or  decide  on  issues  related  to  that  domain”  (Gray,  1928,  p.l  1). 

Collaborating  teams  are  ubiquitous  such  that  the  types  of  tasks  requiring  a  collaborative  effort 
encompass  an  ever-increasing  range  of  situations.  Scientists  collaborate  with  researchers  on  a 
remote  space  station  and  with  robotic  geologists  (Clancy,  2004).  Military  teams  collaborate  to 
develop  plans  as  well  as  during  the  execution  of  those  plans  for  a  wide  variety  of  mission  areas. 
These  teams  all  have  characteristics  in  common  -  they  engage  in  tasks  that  involve  critical 
decisions  and  must  often  coordinate  their  activities  to  accomplish  these  tasks  effectively. 

In  the  following  sections  we  report  on  results  from  a  series  of  studies  where  we  analyzed  data 
obtained  from  teams  and  tasks  characterized  by  the  descriptions  in  the  following  sections. 
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Team  Types 

Teams  of  interest  for  this  research  include  teams  who  employ  asynchronous  or  synchronous 
communications  among  distributed  team  members  to  bring  their  heterogeneous  knowledge  to 
bear  to  solve  the  problem.  Each  team  member  plays  a  functionally  distinct  role  and  contributes 
specialized  knowledge  and  expertise.  Problem-solving  teams  are  often  formed  to  deal  with  a 
rapidly  emerging  difficult  situation  where  consequences  for  error  are  severe.  These  teams  are 
often  ad  hoc  teams  brought  together  in  response  to  a  critical  situation  that  requires  the  expertise 
of  a  diverse  group  of  experts.  The  teams  we  studied  operate  in  complex  socio-technical  settings 
where  the  systems  employed  require  technical  expertise;  teams  operate  within  organizational 
constraints  where  there  are  often  conflicting  goals,  and  the  consequences  for  failure  can  be 
severe. 

Teams  can  be  formally  conceptualized  as  “two  or  more  people  who  interact  dynamically, 
interdepend ently  and  adaptively  toward  a  shared  goal  (Salas,  Dickinson,  Converse  & 
Tannenbaum,  1992,  p.  4).  The  interdependence  involves  all  team  members  sharing  their 
knowledge  about  the  situation,  and  contributing  their  respective  expertise,  to  develop  and 
maintain  an  accurate  understanding  of  the  dynamically  unfolding  situation.  In  many  cases,  teams 
are  also  making  decisions  and  implementing  those  decisions  in  a  timely  manner  to  effectively 
deal  with  the  problem. 

Dynamic  Decision-Making  Tasks 

Dynamic  decision-making  tasks,  such  as  the  decision  domains  investigated  for  this  research,  are 
characterized  by  situations  where:  (1)  A  series  of  decisions  is  needed,  that  is,  the  problem¬ 
solving  event  comprises  many  decisions  to  effectively  deal  with  the  problem  as  it  unfolds,  e.g., 
firefighters,  air  warfare  decision-making,  and  maritime  interdiction  operations  (MIO).  (2) 
Decisions  are  not  independent  because  current  decisions  are  constrained  by  earlier  decisions, 
and,  in  turn  constrain  later  ones.  (3)  The  problem  state  changes  during  the  decision  process  both 
autonomously,  and  as  a  consequence  of  the  decision  maker’s  actions.  (4)  Decisions  are  made  in 
real  time  (Brehmer,  1992).  It  is  necessary  for  the  operator  to  consider  how  the  current  decision 
will  solve  the  immediate  problem,  as  well  as  how  it  will  impact  future  aspects  of  the  overall 
problem-solving  task.  More  importantly,  it  is  not  sufficient  to  make  correct  decisions,  “in  the 
correct  order,  they  also  need  to  be  made  at  the  correct  moment  in  time”  (Brehmer,  1992). 
Dynamic  decision  making  is  inherently  stressful  in  part  because  the  decision  maker  cannot 
control  when  these  critical  decisions  have  to  be  made. 

In  dynamic  decision  making,  decision  making  is  viewed  as  a  form  of  problem  solving,  where  a 
person  seeks  a  viable  course  of  action.  Dynamic  decision  making  tasks  are  found  across  the 
spectrum  of  problem  solving  domains,  including  process  control  plants,  patient  management  in 
hospitals,  managing  a  business,  and  fighting  a  battle.  All  the  tasks  we  examined  were  dynamic 
decision-making  tasks,  as  opposed  to  planning  tasks.  In  Klein’s  (1993)  analysis  of  decision 
errors,  he  refers  to  (decision)  process  errors  and  (decision)  outcome  errors.  Montgomery’s 
approach  (1983,  1989)  views  the  function  of  decisions,  as  “to  prepare  for  action  and  to  make  sure 
that  actions  are  indeed  carried  out”  (Brehmer,  1992,  p.16).  Implementing  the  decision  often 
shapes  both  the  problem  as  well  as  the  cognitive  process  involved  in  decision  making. 

By  implementing  a  decision,  and  obtaining  feedback  on  its  results,  the  operator  has  changed  the 
problem.  For  example,  in  an  air  warfare  scenario,  taking  an  action  against  an  inbound  aircraft, 
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and  observing  that  aircraft’s  response,  or  lack  of  response  to  that  action,  will  recast  the  problem, 
or  change  the  practitioner’s  mental  model  of  that  task,  in  terms  of  detennining  the  intent  of  the 
aircraft.  In  a  similar  vein,  during  a  Coast  Guard  maritime  interdiction  operation  (MIO), 
collecting  new  infonnation  by  obtaining  the  results  of  analysis  of  radiological  data  will  move  the 
boarding  officer’s  assessment  of  the  type  of  cargo  further  along  toward  a  resolution  of  the 
problem  (that  is,  identifying  the  type  of  contraband  cargo  on  the  ship  and  what  type  of  response 
is  required). 

Decision  Making  Situations  are  Part  of  Problem-Solving  Tasks.  We  maintain  that  decision 
making  is  a  critical  element  of  team  problem  solving  when  a  team  is  executing  a  task,  in  contrast 
to  perfonning  a  planning  task.  When  team  members  collaborate  they  make  decisions  and 
implement  those  decisions  as  part  of  executing  their  mission.  Team  communications  often  entail 
asking  or  telling  a  team  member  to  perform  some  action  that  will  move  the  problem  further  along 
toward  completion.  This  contrasts  with  a  team  who  is  collaborating  on  a  planning  task,  where 
deciding  on  and  implementing  decisions  may  not  be  evident.  One  definition  of  a  decision  is  a 
“mental  event  that  occurs  at  a  singular  point  in  time. .  .that  leads  immediately  or  directly  to 
action”  (Hoffman  &  Yates,  2005,  p.  77).  From  this  perspective,  a  decision  is  defined  as  a 
commitment  to  a  course  of  action  (Hoffman  and  Yates,  2005). 

Other  researchers  state  that  decision  making  should  be  thought  of  in  terms  of  deciding,  and  view 
it  as  one  of  a  number  of  macrocognitive  processes  that  it  supports  and  that  support  it.  (Klein, 
Ross,  Moon,  Klein,  Hoffman,  &  Hollnagel,  2003).  Several  factors  impact  deciding,  these 
include  (1)  deciding  involves  instantiating  intentions  and  purposes;  (2)  actions  are  intended  to 
bring  about  states  of  affairs  that  serve  the  interest  of  particular  individuals  or  groups;  (3) 
commitments  to  act  must  be  distinguished  from  action  because,  for  various  reasons,  not  all 
decisions  are  actually  implemented;  and  (4)  choice  among  alternatives  is  never  equivalent  to 
choice  among  consequences,  because  alternatives  rarely  lead  to  single  consequences  (Hoffman  & 
Yates,  2005). 

A  complex  problem-solving  situation  typically  entails  many  decisions,  “either  in  the  process  of 
implementing  a  previous  commitment”  or  the  prior  decision  spawned  new  situations  that 
required  decisions.  This  view  contrasts  with  the  perspective  that  a  decision  signifies  the  end  of  a 
sequence  of  mental  operations  (Hoffman  &  Yates,  2005).  Another  viewpoint  is  of  decisions  as 
discrete  events  with  discrete  decision  points,  however  there  is  not  always  agreement  on  what 
events  are  involved  in  a  given  decision  (Hirokawa  &  Poole,  1996). 

Model  of  Team  Collaboration 

Figure  1  depicts  a  model  developed  to  describe  the  dimensions  of  macrocognition  where  the 
focus  is  on  describing  these  processes  in  a  measurement  framework. 
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{Note:  multiple  overlapping  symbols  indicate  representations  for  multiple  team  members) 

Processes 

_ 


Legend: 


Figure  1.  Measurement  model  for  macrocognition  research,  (from  Fiore,  et  al,  in  press). 


Table  1  includes  descriptions  of  the  macrocognitive  processes  included  in  this  model  of  team 
collaboration,  developed  by  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (in  press).  Individual 
knowledge  building  refers  to  actions  taken  by  individuals  to  build  their  own  knowledge.  Team 
knowledge  building  is  a  process  that  includes  actions  by  teammates  to  disseminate  information 
and  transform  information  into  actionable  knowledge.  Internalized  team  knowledge  refers  to  the 
collective  current  knowledge  held  in  the  individual  minds  of  team  members.  Externalized  team 
knowledge  refers  to  the  current  knowledge  overtly  expressed  to  other  team  members  through 
communication  and/or  artifacts  during  collaborative  team  problem  solving.  Team  problem 
solving  outcomes  are  assessments  of  the  quality  relative  to  a  team’s  problem  solving  or  plan. 
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Table  1.  Definitions  of  Macrocognitive  Processes  Included  in  Model  of  Team  Collaboration. 


Macrocognitive  Process  Categories 

Individual  Knowledge  Building 

Individual  Information  Gathering 

Actions  individuals  engage  in  to  add  to  their  existing  knowledge  such  as 
reading,  asking  questions,  accessing  displays,  etc. 

Individual  Information  Synthesis 

Involves  comparing  relationships  among  information,  context,  and 
artifacts  to  develop  actionable  knowledge. 

Knowledge  Object  Development 

Involves  creation  of  cognitive  artifacts  that  represent  actionable 
knowledge  for  the  task. 

Team  Knowledge  Building 

Team  Information  Exchange 

Passing  relevant  information  to  the  appropriate  teammates  at  the 
appropriate  times 

Team  Knowledge  Sharing 

Explanations  and  interpretations  shared  between  team  members  or  with 
the  team  as  a  whole 

Team  Solution  Option  Generation 

Describes  explanations  and  interpretations  shared  between  team 
members  or  with  the  team  as  a  whole 

Team  Evaluation  and  Negotiation  of 
Alternatives 

Describes  clarifying  and  discussing  the  pros  and  cons  of  potential 
solution  options 

Team  Process  and  Plan  Regulation 

Involves  discussing  or  critiquing  the  team’s  knowledge  building  process 
or  plan  following  feedback  on  its  effectiveness. 

Internalized  Team  Knowledge 

Team  Knowledge  Similarity 

The  degree  to  which  differing  roles  understand  one  another  (e.g.,  how 
well  a  land/sea  vehicle  specialist  understands  a  humanitarian  specialist), 
or  how  well  the  team  members’  understand  the  critical  goals  and 
locations  of  important  resources  (shared  situation  awareness). 

Team  Knowledge  Resources 

Team  members’  collective  understanding  of  resources/  responsibilities 
associated  with  the  task. 

Inter-positional  Knowledge 

Accurate  knowledge  regarding  position-specific  roles,  goals, 
responsibilities,  access  to  information,  constraints,  and  interdependen¬ 
cies  with  other  team  positions. 

Individual  Situational  Awareness 

Accurate  awareness  of  moment  to  moment  changes  in  the  team’s 
environment.  The  construct  has  been  defined  previously  by  Endsley 
(1995) 

Externalized  Team  Knowledge 

Externalized  Cue-Strategy 

Association 

Describes  the  team’s  collective  agreement  as  to  their  task  strategies 
and  the  situational  cues  that  modify  those  strategies  (and  how). 

Pattern  Recognition  and  Trend 
Analysis 

Refers  to  the  accuracy  of  the  patterns  or  trends  explicitly  noted  by 
members  of  a  team  that  is  either  agreed  upon  or  unchallenged  by 
other  team  members. 

Uncertainty  Resolution 

The  degree  to  which  a  team  has  collectively  agreed  upon  the  status  of 
problem  variables  (e.g.,  hostile/friendly). 

Team  Problem  Solving  Outcomes 

Quality  of  plan  (problem  solving 
solution) 

Involves  the  degree  to  which  the  solution  adopted  by  a  problem  solving 
team  achieves  a  resolution  to  the  problem  (e.g.,  limit  fatalities,  limit 
destruction). 

Efficiency  of  planning  process 

Amount  of  time  it  takes  a  problem  solving  team  to  arrive  at  a 
successful  resolution  to  a  problem. 

Efficiency  of  plan  execution 

Quality  of  the  plan  (e.g.,  number  of  lives  saved)  divided  by  the  amount 
of  resources  used  to  accomplish  this  and  the  amount  of  time  the  plan 
takes  to  unfold. 
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Method 


Coding  Process 

Verbatim  transcripts  or  chat  logs  were  analyzed  from  an  Air  Force  exercise  and  one  real-world 
event  where  teams  collaborated  to  solve  a  complex  problem:  a  dynamic  planning  and  execution 
exercise  involving  time-sensitive  targeting  and  the  communications  that  transpired  between  the 
personnel  at  the  North  American  Aerospace  Defense  Command  (NORAD),  on  Sept.  11,  2001. 
Transcripts  included  communications  that  occurred  between  all  team  members  as  well  as  with 
decision  makers  at  the  distributed  sites.  Team  communications  data  were  analyzed  and  coded 
using  the  definitions  of  the  macrocognitive  processes  in  the  model  of  team  collaboration, 
developed  by  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (in  press).  Coders  practiced  on  a 
separate  set  of  team  communications  and  calibrated  their  coding  after  coding  200  lines.  The  two 
coders  reviewed  their  coding  with  one  of  the  authors  and  discussed  any  differences  in 
interpreting  the  definitions  prior  to  coding  the  communications  data  to  be  analyzed.  Analysis  of 
the  Air  Operations  Center  data  and  the  NORAD  data  comprised  2515  lines,  and  1517  lines  of 
coding,  respectively. 

Problem-Solving  Tasks 

We  analyzed  team  communications  from  two  decision-making  scenarios  described  in  the 
following  paragraphs. 

Air  Operations  Center.  A  training  exercise  in  dynamic  planning  operations  was  conducted  at 
the  Air  Force  Air  Operations  Center  (AOC),  in  October  2008.  This  Training  Research  Exercise 
was  conducted  to  employ  operational  concepts  and  training  techniques  from  several  commands. 
Participants  from  the  USAF  Warfare  Center,  Naval  Strike  Air  Warfare  Center,  Special  Opera¬ 
tions,  and  US  Army  worked  together  on  dynamic  planning  operations.  Team  collaboration 
communication  was  recorded  in  Microsoft  Internet  Relay  Chat  (mIRC)  logs  across  15  different 
chat  rooms,  comprising  twelve  hours  of  exercise  data  that  was  recorded  in  six  two-hour  segments. 

NORAD.  On  Sept  11,  2001,  air  traffic  controllers  in  New  York,  Boston,  Washington,  and 
Cleveland  were  scrambling  due  to  the  hijacking  of  four  American  commercial  airliners.  In  their 
efforts  to  bring  order  to  chaos  the  Federal  Aviation  Administration  (FAA)  in  communication 
with  Northeast  Air  Defense  Sector  (NEADS)  and  North  American  Aerospace  Defense  Command 
(NORAD)  scrambled  fighter  aircraft  to  escort  the  airliners.  Transcripts  of  the  team  collaboration 
communications  that  transpired  to  bring  all  aircraft  down  on  September  11,  2001,  in  response  to 
the  terrorist  attack  were  analyzed. 

RESULTS 

Table  2  presents  the  percentage  of  speech  turns  coded  as  representing  the  macrocognitive 
processes  included  in  the  model  of  team  collaboration.  The  most  significant  finding  is  that  a  new 
macrocognitive  process  emerged  during  the  coding  process:  Decision  to  take  action.  Two  types 
of  decisions  were  evident:  issuing  a  course  of  action  and  requesting  a  peer  to  take  action.  These 
two  types  of  decisions  are  differentiated  by  the  authority  relationship  between  the  speaker  and 
receiver  and  the  criticality  of  the  action  to  be  taken.  A  course  of  action  is  issued  by  a  superior  to 
a  subordinate  and  the  action  to  be  implemented  is  more  critical  to  the  outcome  of  the  scenario. 
Requesting  a  peer  to  take  action  involves  a  lower  level  type  of  decision  in  terms  of  the  authority 
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relationship,  that  is,  peer-to-peer,  and  the  less  critical  nature  of  the  action  to  the  outcome  of  the 
scenario. 


Table  2.  Percentage  of  Macrocognitive  Processes  used  Across  Decision-Making  Domains. 


Code 

Macrocognitive  Process  Categories 

Percentage  of  Speech  Turns 

Individual  Knowledge  Building 

Air  Ops  Center 

NORAD 

IIG 

Individual  Information  Gathering 

16.66 

29.37 

IIS 

Individual  Information  Synthesis 

1.04 

1.66 

KOB 

Knowledge  Object  Development 

0.00 

0.00 

Team  Knowledge  Building 

TIE 

Team  Information  Exchange 

37.57 

50.44 

TKS 

Team  Knowledge  Sharing 

5.45 

3.58 

TSOG 

Team  Solution  Option  Generation 

0.35 

2.93 

TENA 

Team  Evaluation  and  Negotiation  of  Alternatives 

0.13 

0.00 

TPPR 

Team  Process  and  Plan  Regulation 

0.00 

0.00 

Internalized  Team  Knowledge 

ITK 

Team  Knowledge  Similarity 

0.03 

0.00 

TKR 

Team  Knowledge  Resources 

0.06 

0.00 

IK 

Inter-positional  Knowledge  (3) 

0.06 

0.19 

ISA 

Individual  Situational  Awareness  (1) 

0.00 

1.60 

Externalized  Team  Knowledge 

ECSA 

Externalized  Cue-Strategy  Association 

0.13 

0.06 

PRTA 

Pattern  Recognition  and  Trend  Analysis 

0.11 

0.06 

UR 

Uncertainty  Resolution 

0.00 

0.12 

Problem  Solving  Outcomes 

QOP 

Quality  of  plan  (problem  solving  solution) 

0.00 

0.00 

EPP 

Efficiency  of  planning  process 

0.00 

0.00 

EPE 

Efficiency  of  plan  execution 

0.00 

0.00 

Decision  to  Take  Action 

DTA:  COA 

Superior  to  subordinate:  issuing  a  course  of  action 

4.72 

1.21 

DTA:  RTA 

Peer-to-peer  request  a  team  member  take  action 

2.75 

4.09 

In  general,  the  large  number  of  speech  tums/chat  log  entries  coded  as  individual  information 
gathering  and  team  infonnation  exchange  indicate  the  huge  emphasis  knowledge  construction 
that  is  required  for  both  tasks.  The  results  reported  here  represent  an  initial  analysis;  additional 
analysis  is  planned. 

Problem  Solving  Includes  Taking  Actions 

These  results  corroborate  results  from  other  analyses  based  on  analysis  and  coding  transcripts 
from  other  dynamic  decision-making  tasks  (Hutchins  and  Kendall,  in  press;  Hutchins  and 
Kendall,  2008).  For  example,  in  a  30-minute  air  warfare  scenario,  nine  decisions  to  take  action 
were  made  by  the  air  warfare  team.  Nineteen  percent  of  the  team  communications  from  the  Fire 
Department  of  New  York  responding  to  the  Sept  11,  2001,  attacks  on  the  World  Trade  Center 
were  decisions  to  take  action. 
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Table  3  provides  examples  of  the  macrocognitive  process  Decision  to  Take  Action  from  the 
NORAD  data.  It  includes  examples  of  two  sub-categories:  issuing  an  order  regarding  a  course  of 
action  (coa)  and  requesting  a  team  member  take  some  action  (rta). 

Table  3.  Examples  of  Decision  to  Take  Action  from  NORAD  Coding. 


Speech  Turn 

Code 

Your  pilots  should  be  loading  and  just  make  sure  your  pilots  load  up  their  mode  2  and  mode  4. 

DTA  (COA) 

Give  me  a  track  number  on  that  bomb  —  that  guy  going  by  Cleveland. 

DTA  (RTA) 

Give  me  an  arrow,  Bud.  Scope  2,  scope  1 . 

DTA  (RTA) 

Just  make  sure  its  squawking. 

DTA  (RTA) 

On  your  mode  2,  make  sure  that’s  standard  and  also  make  sure  you’re  mode  4  is  all 
loaded  up  as  well. 

DTA  (COA) 

Turn  him  around  and  have  him  go  look. 

DTA  (RTA) 

Can  you  help  some  of  these  people  at  tracking  this  bird? 

DTA  (RTA) 

And  get  all  mode  3. 

DTA  (COA) 

Have  them  call  Washington  Center  for  that. 

DTA  (RTA) 

If  you  don’t  see  them,  call  right  away.  If  you  see  it  and  they  haven’t  hit  it  up,  call  that  center. 

DTA  (COA) 

Tell  them  we  need  to  know  where  Air  Force  One  is. 

DTA  (COA) 

Table  4  provides  examples  from  the  air  operations  center  Chat  log  exchanges  coded  as  macro- 
cognitive  processes  in  the  model  of  team  collaboration. 


Table  4.  Examples  from  Chat  log  Entries  Coded  as  Macrocognitive  Processes  in  the 

Model  of  Team  Collaboration. 


Additional  Codes 

1 

Administration 

■  I  was  told  not  to  be  in  Special  Operation  Liaison  Element  (SOLE)  room 

■  Deed  how  copy? 

■  Test 

■  Battlefield  Coordination  Detachment  (BCD)  Fires  do  you  have  communication  with  the  CHOPS? 

Miscellaneous 

■  Have  Information  Operation  (IO)  tell  (A/C  #9)  that  we’ll  have  some  jack  in  the  crack  waiting  for  him 

■  Roger  (RGR)  Thank  You 

■  CO  rgr  on  *TARGET* 

COA:  Course  of  Action 

■  Contact  (a/c  #12)  on  TAD  (circuit  #2)  for  clearance  to  drop. 

■  Intelligence  Surveillance  and  Reconnaissance  Cell  (ISRC):  Move  a/c  to  provide  over  watch  for 

Special  Operation  Force  (SOF)  teams  at  *TARGET* 

■  Decsole:  you  can  move  both  A/C  #2  and  A/C  #10  to  training  camp  loc  vicinity  (city  #3)  and  (city  #5) 
to  Bakersfield.  Upon  completion  of  mission,  and  NLT  1 102Z,  both  assets  return  to  current  location 

RTA:  Request  to  Take  Action 

■  Please  pass  reports  in  Intel  Reports 

■  Analysis  Correlation  and  Fusion  (ACF):  Pass  coordinates  for  (Location)  Transload 

Individual  Knowledge  Building 
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2 

IIG:  Individual  Information  Gathering 

■  QUESTIONS:  What  is  the  correct  way  to  pass  tasking  to  a  predator  to  attack? 

■  Joint  Coordinating  Element  (JCE)  do  you  know  the  local  threat/  risk  in  (target  location),  and  do  you 
have  imagery  location  of  the  locations? 

■  Any  Battle  Damage  Assessment  (BDA)  reports/imagery  post-strike  for  aircraft? 

IIS:  Individual  Information  Synthesis 

■  CO  -  Negative  -  Tomahawk  Land  Attack  Missile  (TLAM)  only  hitch  is  in  SEAL  teams  burn  them  for 
ground  support. 

■  Senior  Intelligence  Duty  Officer  (SIDO),  Reliable  sources  report  probably  radiological  transload  site 
at  @LOC ATION @ 

■  Reliable  sources  report  known  (Country  #2)  bomb  component  supplier,  is  awaiting  a  large  shipment 
of  explosives 

■  It  is  suspected  that  (Country  #1)  uses  this  location  as  a  storage  facility  for  spent  fuel. 

KOD:  Knowledge  Object  Development 

■  Code  not  used 

Team  Knowledge  Building 

3 

TIE:  Team  Information  Exchange 

•  *****  priority  coordinated,  entered  and  pushed  to  Joint  Time  Sensitive  Targeting  Manager  (JTSTM) 

■  The  actual  snatch  and  grab  would  be  possibility  for  SOF  but  we  would  need  the  intelligence  assist. 

■  ISRC:  for  your  information,  @LOCATION@  is  now  SOF  mission;  reconnaissance  a/c  #1  to  provide 
over  watch,  SOF  is  in  contact  with  a/c  now. 

TKS:  Team  Knowledge  Sharing 

■  Self  defense  applies  for  hostile  acts  from  (Country  #3)  fighters  in  (Country  #  2)  or  (Country  #4) 
airspace 

■  Enemy  forces  that  employ  ordnance,  Electronic  Attack  (EA),  or  fore  control  systems  (achieve  a  radar 
lock)  against  friendly  forces  have  committed  a  hostile  act. 

TSOG:  Team  Solution  Option  Generation 

■  Awaiting  radiological  impact  assessment  on  watershed  if  strike  building.  Second  option  in  work  is 
deny  (destroy)  local  roads  to  prevent  access  in/out. 

■  If  we  crater  the  runway  and  taxiways,  we  may  be  able  to  effectively  stop  the  target. 

■  To  shorten  timeline  for  tactical  tomahawk  we  can  launch  to  loiter. 

■  Will  attempt  to  mitigate  with  weaponeering 

TENA:  Team  Evaluation  and  Negotiation  of  Alternatives 

■  Target  Duty  Officer  (TDO):  Just  throwing  this  out  there,  but  if  you  target  the  roadways,  is  there  a 
chance  you  could  spook  them  and  they  might  fire  off  their  missiles  and  run? 

TPPR:  Team  Process  and  Plan  Regulation 

■  Not  used 

Internalized  Team  Knowledge 

4 

TKS:  Team  Knowledge  Similarity 

■  SIMISM.  C2WSPTT  always  expends  all  weapons  on  attack 

■  He  wouldn't  request  return  to  base  he’d  tell  you  he  is  returning  to  base 

TKR:  Team  Knowledge  Resources 

■  Internositional  Knowledge  (IK):  I  remember  sketchv  authentication 

■  Individual  situation  awareness  (ISA):  a/c  #2  is  out  of  position,  looks  like  other  strike  assets  quicker 

Externalized  Team  Knowledge 

5 

ECSA:  Externalized  Cue-strategy  Association 

■  The  DEC  Chief  stated  that  if  there  is  an  erect  launcher  in  a  JSOA  his  "ROE"  is  to  kill  it  ASAP  and  if 
there  is  time  to  deconflict  with  the  teams 

■  He  mentioned  TLAMs  wouldn't  be  deconflicted  either,  but  I  dispute  that  logic.  First,  we  wouldn't  use 
a  TLAM  shot  to  kill  a  launcher  I  don't  think.  Unless  it  was  a  last  resort. 

■  Can  get  SOF  Team  to  location  as  additional  resource  if  we  elect  to  monitor  the  site  for  any  potential 
leadership  meetings  that  may  occur  later  given  that  location  is  used  for  meetings  and  (target)  is  there 
now. 
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PRTA:  Pattern  Recognition  and  Trend  Analysis 

■  Co,  TDO,  looks  like  ******  may  be  similar  to  our  first  target  with  regards  to  unknown  presence  of 
Radiological  containers  in  facility.  We  would  look  at  interdiction  for  containment  to  prevent  travel 
to/fm  that  site,  your  thoughts  on  best  plan/option 

UR:  Uncertainty  Resolution 

■  TLAMs  most  definitely  have  to  be  de-conflicted  even  for  over  flight  of  the  Joint  Special  Operations 
Area  (JSOA)-unless  direct  otherwise  by  the  Joint  Force  Commander  (JFC) 

Team  Problem  Solving  Outcomes 

6 

QOP:  Quality  of  Plan 

•  Not  used 

EPP:  Efficiency  of  Planning  Process 

•  Not  used 

EPE:  Efficiency  of  Plan  Execution 

•  Not  used 

DISCUSSION 

A  new  macrocognitive  process  emerged  during  the  coding  process:  Decision  to  take  action. 
Deciding  to  take  action  is  viewed  as  both  a  macrocognitive  process  and  a  product  of  team 
collaboration.  Results  indicate  additional  macrocognitive  processes  need  to  be  included  in  the 
model  to  represent  decision  making  which  occurs  during  execution  of  real-world  tasks. 

New  Coding  Category:  Decision  to  Take  Action 

Many  critical  tasks  that  involve  team  collaboration  include  team  members  taking  action  in 
addition  to  developing  new  knowledge  and  agreeing  on  a  final  solution.  Various  actions  are 
taken  as  part  of  the  overall  information  gathering  process  (e.g.,  MIOs,  air  warfare,  firefighters, 
etc.).  Moreover,  dynamic  decision-making  tasks  entail  a  series  of  decisions  as  part  and  parcel  of 
problem  solving.  Many  tasks  involve  an  interleaving  of  knowledge  building,  decision  making 
and  taking  action  in  order  to  accomplish  the  mission.  For  example,  in  MIOs  members  of  the 
boarding  party  physically  search  the  ship  using  sensing  equipment  to  take  various  readings 
which  are  sent  to  experts  at  reachback  centers  for  analysis.  In  some  cases,  members  of  the 
boarding  party  are  then  asked  to  take  additional  readings  to  provide  more  fine-grained  data  that 
will  help  more  precisely  determine  the  type  of  cargo  on  the  ship.  This  process  of  physically 
searching  the  ship  for  contraband  cargo  and  suspect  people  entails  a  series  of  actions,  as  reflected 
in  many  speech  turns  requesting  a  team  member  to  take  an  action. 

For  the  task  domains  reported  on  here,  a  constant  interplay  exists  between  sharing  information  to 
develop  new  knowledge  and  maintain  situation  awareness  and  then  executing,  or  implementing 
actions,  followed  by  monitoring  and  building  new  knowledge  on  the  unfolding  situation. 
Execution  of  the  mission,  or  problem-solving  problem,  would  come  to  a  screeching  halt  without 
this  continual,  iterative  cycle  of  developing  knowledge  of  the  situation  and  responding  to  the 
current  situation  by  taking  various  actions  that  move  the  problem  along. 

Challenges  associated  with  analyzing  this  type  of  data  include:  (1)  the  need  to  be  familiar  with 
the  domain  to  understand  processes  employed  and  jargon;  (2)  speech  turns  cannot  be  coded  in 
isolation  -  following  the  many  different  “threads”  being  discussed,  which  are  inter-leaved  with 
other  threads,  necessitates  going  back  through  the  transcript  to  follow  the  thought  process  for 
each  of  the  various  sub-tasks;  (3)  resolving  differences  in  coding  is  a  collaborative  process  where 
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each  person  expresses  their  interpretation  of  the  speech  turn;  (4)  in  some  cases,  needing  to  listen 
to  the  audio  for  intonation;  (5)  in  some  cases  we  accessed  subject  matter  experts  to  decipher 
jargon  and  understand  standard  operating  procedures;  and  (6)  one  needs  to  understand  the 
relationships  between  team  members  from  different  organizations  and  the  hierarchy  of  the 
organizations. 

Several  macrocognitive  processes  included  in  the  model  were  not  evident  in  the  two  decision¬ 
making  domains  analyzed.  This  is  attributed  to  the  perspective  used  when  developing  definitions 
for  these  processes,  which  was  from  a  measurement  perspective.  Measures  lean  heavily  toward 
laboratory  experiments  so  several  are  not  possible  when  analyzing  transcripts  of  real-world 
scenarios.  We  typically  obtain  the  transcripts  after  the  fact,  such  as,  the  Fire  Department  of  New 
York  collaborating  to  deal  with  the  terrorist  attack  on  Sept  11,  2001,  NORAD/FAA  collaborating 
to  get  all  aircraft  out  of  the  sky,  also  on  9-1 1,  etc.  so  we  do  not  have  the  ability  to  ask  participants 
to  complete  surveys  or  obtain  many  of  the  measures  included  for  the  cognitive  processes  in  this 
model  of  team  collaboration. 

CONCLUSIONS 

A  new  macrocognitive  process  emerged  during  the  coding  process:  decision  to  take  action. 
Results  indicate  additional  macrocognitive  processes  need  to  be  included  in  the  model  to 
represent  decision  making  which  occurs  during  execution  of  real-world  tasks.  Deciding  to  take 
action — in  the  dynamic  decision-making  domains  analyzed  for  this  research — is  viewed  as  both 
a  macrocognitive  process  and  a  product  of  team  collaboration.  Results  indicate  additional 
macrocognitive  processes  need  to  be  included  to  the  model  to  represent  decision  making  which 
occurs  during  execution  of  real-world  tasks. 

Many  real-world  tasks  do  not  have  criteria  for  assessing  problem-solving  outcomes,  such  as, 
MIO,  air  warfare,  firefighters,  NORAD,  and  the  AOC.  For  the  tasks  analyzed  to  date  for  this 
research,  no  metrics  are  available  to  assess  the  “goodness”  of  the  team’s  performance,  thus,  there 
was  no  evidence  for  several  of  the  macrocognitive  processes  included  in  the  model.  In  a  similar 
vein,  speed  is  not  always  a  criterion  for  good  performance,  such  as  planning  at  the  operational 
level.  When  planning  at  the  operational  level  of  war,  the  focus  is  more  on  the  quality  of  the  plan 
versus  time  to  build  the  plan.  In  several  cases,  these  metrics  do  not  transfer  to  measuring  the 
macrocognitive  processes  entailed  in  real-world  problem-solving  situations. 

Through  use  of  a  cognitive  systems  engineering  approach  researchers  can  gain  insight  into 
cognitive  functions  used  by  team  members  and  how  teams  perform  these  functions.  This  insight 
can  then  be  used  to  support  the  cognitive  requirements  of  work,  in  this  case,  team  collaboration 
activities.  The  research  reported  here  represents  one  aspect  of  a  larger  effort  with  the  long-tenn 
goal  of  contributing  to  the  design  of  collaborative  tools  to  provide  better  support  to  collaborating 
teams. 
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Macocognition 

•  Mental  activities  that  must  be  successfully  accomplished  to  perform  a 
task  or  achieve  a  goal  (Klein,  Ross,  Moon,  Klein,  Hoffman,  &  Hollnagell,  2003). 

•  The  internalized  and  externalized  high-level  mental  processes 
employed  by  teams  to  create  new  knowledge  during  complex,  one-of-a- 
kind  problem  solving. 

•  Functions  are  generally  performed  during  collaborative  team  problem 
solving,  where  the  emphasis  is  on  building  new  knowledge. 

•  A  complex,  multi-level  phenomenon  that  involves  development, 
refinement,  and  maintenance  of  higher-order  cognitive  processes  and 
emergent  states  (Burke,  2007). 

•  Cognitive  processes  employed  by  team  members  in  unique, 
information-rich,  time-compressed  collaborative  problem  solving 

-  Individual  and  team  knowledge  development 

-  Shared  problem  conceptualization 

-  Mental  model  development 

-  Solution  option  generation 

-  Detecting  problems,  developing  and  sharing  situation  awareness, 
generating  options,  using  analogues,  mentally  simulating  courses  of 
action,  planning  and  re-planning,  maintaining  vigilance,  and  assessing 
risk  (Klein,  2001). 


Team  Collaboration 


Collaboration  occurs  “when  a  group  of  autonomous 

stakeholders  of  a  problem  domain  engage  in  an  interactive 
process,  using  shared  rules,  norms,  and  structures  to  act  or 
decide  on  issues  related  to  that  domain”  (Gray,  1998,  p.ii). 

-  Provides  increased  information  processing  capacity  where  more 
minds  are  enlisted  to  handle  complex  problems  (Hocevar,  Jansen, 
and  Thomas,  2004). 


-  Team  members  provide  several  perspectives  on  an  issue  for 
generating,  choosing,  and  implementing  action  plans. 


-  A  collaborative  approach  also 
provides  greater  flexibility  and 
innovation  where  human 
judgment  and  experience  are 
leveraged  (Hocevar,  et  al,  2004.) 


as  committees  or  teams 


Coalition  Partners 


Network 

Operation 

Center 


^  Austria 

Sweden 

^  Singapore 


Represents  Decision 

Making  Nodes 

r\ 

Contributor-Team 

u 

member  of  Node 

Team  Types 


[SS9  NAVAL 

’Cru.i.iiim.ii.i.wjT  POSTGRADUATE 
SCHOOL 


Teams  who  employ  asynchronous  or  synchronous  communications 
among  distributed  team  members  to  bring  their  heterogeneous 
knowledge  to  bear  to  solve  the  problem. 

-  Each  team  member  plays  a  functionally  distinct  role  and  contributes 
specialized  knowledge  and  expertise. 

-  Problem-solving  teams  are  often  formed  to  deal  with  a  rapidly 
emerging  difficult  situation  where  consequences  for  error  are  severe. 

-  Ad  hoc  teams  brought  together  in  response  to  a  critical  situation  that 
requires  the  expertise  of  a  diverse  group  of  experts. 

-  Operate  in  complex  socio-technical  settings  where  the  systems 
employed  require  technical  expertise 

-  Operate  within  organizational  constraints  where  there  are  often 
conflicting  goals 


===^— 


y~  YVf  fTTT 

,  _ _  v 

Hill  ^  J 

Dynamic  Decision-Making  Tasks 

•  A  series  of  decisions  is  needed,  that  is,  the  problem-solving  event 
comprises  many  decisions  to  effectively  deal  with  the  problem  as  it 
unfolds,  e.g.,  firefighters,  air  warfare  decision-making,  and  maritime 
interdiction  operations  (MIO). 

•  Decisions  are  not  independent  because  current  decisions  are 
constrained  by  earlier  decisions,  and,  in  turn  constrain  later  ones. 

•  The  problem  state  changes  during  the  decision  process  both 
autonomously,  and  as  a  consequence  of  the  decision  maker’s  actions. 

•  Decisions  are  made  in  real  time  (Brehmer,  1992). 

-  It  is  necessary  for  the  operator  to  consider  how  the  current 
decision  will  solve  the  immediate  problem,  as  well  as  how  it  will 
impact  future  aspects  of  the  overall  problem-solving  task. 

-  It  is  not  sufficient  to  make  correct  decisions,  “in  the  correct  order, 
they  also  need  to  be  made  at  the  correct  moment  in  time” 
(Brehmer,  1992). 

-  Dynamic  decision  making  is  inherently  stressful  in  part  because 
the  decision  maker  cannot  control  when  these  critical  decisions 
have  to  be  made. 
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Dynamic  Decision-Making  Tasks  (cont  d) 

•  Decision  making  is  viewed  as  a  form  of  problem  solving,  where 
a  person  seeks  a  viable  course  of  action. 

•  Dynamic  decision  making  tasks  are  found  across  the  spectrum 
of  problem  solving  domains,  including  process  control  plants, 
patient  management  in  hospitals,  managing  a  business,  and 
fighting  a  battle.  All  the  tasks  we  examined  were  dynamic 
decision-making  tasks,  as  opposed  to  planning  tasks. 

•  In  Klein’s  (1993)  analysis  of  decision  errors,  he  refers  to 
(decision)  process  errors  and  (decision)  outcome  errors. 

•  Montgomery’s  approach  (1983,  1989)  views  the  function  of 
decisions,  as  “to  prepare  for  action  and  to  make  sure  that 
actions  are  indeed  carried  out”  (Brehmer,  1992,  p.16). 

•  Implementing  the  decision  often  shapes  both  the  problem  as 
well  as  the  cognitive  process  involved  in  decision  making. 
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Model  of  Team  Collaboration  (From  Fiore,  Smith-Jentsch,  Salas,  Warner, 


&  Letsky  (2008) 
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Definitions  of  Macrocog nitive  Processes  Included  in 

Model  of  Team  Collaboration 

(From  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (2008) 


Macrocognitive  Process  Categories 

Individual  Knowledge  Building 

Individual  Information  Gathering 

Actions  individuals  engage  in  to  add  to  their  existing  knowledge 
such  as  reading,  asking  questions,  accessing  displays,  etc. 

Individual  Information  Synthesis 

Involves  comparing  relationships  among  information,  context,  and 
artifacts  to  develop  actionable  knowledge. 

Knowledge  Object  Development 

Involves  creation  of  cognitive  artifacts  that  represent  actionable 
knowledge  for  the  task. 

Team  Knowledge 

Team  Information  Exchange 

Passing  relevant  information  to  the  appropriate  teammates  at  the 
appropriate  times 

Team  Knowledge  Sharing 

Explanations  and  interpretations  shared  between  team  members  or 
with  the  team  as  a  whole 

Team  Solution  Option  Generation 

Describes  explanations  and  interpretations  shared  between  team 
members  or  with  the  team  as  a  whole 

Team  Evaluation  and  Negotiation  of 
Alternatives 

Describes  clarifying  and  discussing  the  pros  and  cons  of  potential 
solution  options 

Team  Process  and  Plan  Regulation 

Involves  discussing  or  critiquing  the  team’s  knowledge  building 
process  or  plan  following  feedback  on  its  effectiveness. 

Definitions  of  Macrocognitive  Processes  Included  in 

Model  of  Team  Collaboration 

(From  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (2008) 


Internalized  Team  Knowledge 

Team  Knowledge  Similarity 

The  degree  to  which  differing  roles  understand  one  another  (e.g.,  how  well  a 
land/sea  vehicle  specialist  understands  a  humanitarian  specialist),  or  how  well  the 
team  members’  understand  the  critical  goals  and  locations  of  important  resources 
(shared  situation  awareness). 

Team  Knowledge  Resources 

Team  members’  collective  understanding  of  resources/  responsibilities  associated 
with  the  task. 

Inter-positional  Knowledge 

Accurate  knowledge  regarding  position-specific  roles,  goals,  responsibilities,  access 
to  information,  constraints,  and  interdependencies  with  other  team  positions. 

Individual  Situational 

Awareness 

Accurate  awareness  of  moment  to  moment  changes  in  the  team’s  environment.  The 
construct  has  been  defined  previously  by  Endsley  (1995) 

Externalized  Team  Knowledge 

Externalized  Cue-Strategy 
Association 

Describes  the  team’s  collective  agreement  as  to  their  task  strategies  and  the 
situational  cues  that  modify  those  strategies  (and  how). 

Pattern  Recognition  and 
Trend  Analysis 

Refers  to  the  accuracy  of  the  patterns  or  trends  explicitly  noted  by  members  of  a 
team  that  is  either  agreed  upon  or  unchallenged  by  other  team  members. 

Uncertainty  Resolution 

The  degree  to  which  a  team  has  collectively  agreed  upon  the  status  of  problem 
variables  (e.g.,  hostile/friendly). 

Definitions  of  Macrocognitive  Processes  Included  in 

Model  of  Team  Collaboration 

(From  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (2008) 


Team  Problem  Solving  Outcomes 

Quality  of  plan  (problem 
solving  solution) 

Involves  the  degree  to  which  the  solution  adopted  by  a  problem 
solving  team  achieves  a  resolution  to  the  problem  (e.g.,  limit 
fatalities,  limit  destruction). 

Efficiency  of  planning 
process 

Amount  of  time  it  takes  a  problem  solving  team  to  arrive  at 
a  successful  resolution  to  a  problem. 

Efficiency  of  plan 
execution 

Quality  of  the  plan  (e.g.,  number  of  lives  saved)  divided  by  the 
amount  of  resources  used  to  accomplish  this  and  the  amount  of 
time  the  plan  takes  to  unfold. 
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Method 

•  Verbatim  transcripts/chat  logs  were  analyzed  from  an  Air  Force  exercise 
and  one  real-world  event  where  teams  collaborated  to  solve  a  complex 
problem 

-  Air  Force  Air  Operations  Center  TREX:  Dynamic  planning  and 
execution  exercise  involving  time-sensitive  targeting 

-  North  American  Aerospace  Defense  Command  (NORAD),  on  Sept. 

11,2001 

•  Transcripts  included  communications  that  occurred  between  all  team 
members  as  well  as  with  decision  makers  at  the  distributed  sites. 

•  Team  communications  data  were  analyzed  and  coded  using  definitions 
of  the  macrocognitive  processes  in  the  model  of  team  collaboration, 
developed  by  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (2008). 

•  Coders  practiced  on  a  separate  set  of  team  communications  and 
calibrated  their  coding  after  coding  200  lines. 

-  Two  coders  reviewed  their  coding  with  one  of  the  authors  and 
discussed  any  differences  in  interpreting  the  definitions  prior  to 
coding  the  communications  data  to  be  analyzed. 

-  Air  Operations  Center  data:  2515  lines  of  code 

-  NORAD  data:  1517  lines  of  code 

-  Each  utterance  was  coded  separately 


Percentage  of  Macrocognitive  Processes  used  Across  Decision-Making  Domains 


Code 

Macrocognitive  Process  Categories 

Percentage  of  S 

jeech  Turns 

Individual  Knowledge  Building 

Air  Ops  Center 

NORAD 

IIG 

Individual  Information  Gathering 

16.66 

29.37 

IIS 

Individual  Information  Synthesis 

1.04 

1.66 

KOB 

Knowledge  Object  Development 

0.00 

0.00 

Team  Knowledge  Building 

TIE 

Team  Information  Exchange 

37.57 

50.44 

TKS 

Team  Knowledge  Sharing 

5.45 

3.58 

TSOG 

Team  Solution  Option  Generation 

0.35 

2.93 

TENA 

Team  Evaluation  and  Negotiation  of  Alternatives 

0.13 

0.00 

TPPR 

Team  Process  and  Plan  Regulation 

0.00 

0.00 

Internalized  Team  Knowledge 

ITK 

Team  Knowledge  Similarity 

0.03 

0.00 

TKR 

Team  Knowledge  Resources 

0.06 

0.00 

IK 

Inter-positional  Knowledge  (3) 

0.06 

0.19 

ISA 

Individual  Situational  Awareness  (1) 

0.00 

1.60 

Externalized  Team  Knowledge 

ECS  A 

Externalized  Cue-Strategy  Association 

0.13 

0.06 

PRTA 

Pattern  Recognition  and  Trend  Analysis 

0.11 

0.06 

Uncertainty  Resolution 

0.00 

0.12 

Problem  Solving  Outcomes 

QOP 

Quality  of  plan  (problem  solving  solution) 

0.00 

0.00 

EPP 

Efficiency  of  planning  process 

0.00 

0.00 

EPE 

Efficiency  of  plan  execution 

0.00 

0.00 

Decision  to  Take  Action 

DTA:  COA 

To  subordinate:  issuing  a  course  of  action 

4.72 

1.21 

DTA:  RTA 

Peer-to-neer  request  a  team  member  take  action 

2.75 

4.09 

Problem  Solving  Includes  Taking  Actions 


Examples  of  Decision  to  Take  Action  from  NORAD  Coding 

Decisions 

Your  pilots  should  be  loading  and  just  make  sure  your  pilots  load  up  their  mode  2  and 
mode  4. 

COA 

Give  me  a  track  number  on  that  bomb  —  that  guy  going  by . 

RTA 

Give  me  an  arrow,  Bud.  Scope  2,  scope  1. 

RTA 

Just  make  sure  its  squawking. 

RTA 

On  your  mode  2,  make  sure  that’s  standard  and  also  make  sure  you’re  mode  4  is  all 
loaded  up  as  well. 

COA 

Turn  him  around  and  have  him  go  look. 

RTA 

Can  you  help  some  of  these  people  at  tracking  this  bird? 

RTA 

And  get  all  mode  3. 

COA 

Have  them  call  for  that. 

RTA 

If  you  don’t  see  them,  call  right  away.  If  you  see  it  and  they  haven’t  hit  it  up, 
call  that  center. 

COA 

Tell  them  we  need  to  know  where  Air  Force  One  is. 

COA 

Chat  log  Entries  Coded  as  Macrocognitive  Processes  in  the 

Model  of  Team  Collaboration 


Individual  Knowledge  Building 

IIG:  Individual  Information  Gathering 

■What  is  the  correct  way  to  pass  tasking  to  a  predator  to  attack? 

■Do  you  know  the  local  threat/  risk  in  (target  location),  and  do  you  have  imagery  location  of  the  locations 

IIS:  Individual  Information  Synthesis 

■Senior  Intelligence  Duty  Officer  (SIDO),  Reliable  sources  report  probably  radiological  transload  site  at 
@LOC  ATION  @ 

■It  is  suspected  that  (Country  #1)  uses  this  location  as  a  storage  facility  for  spent  fuel. 

fCOD:  Knowledge  Object  Development 

■Code  not  used 

Team  Knowledge  Building 

TIE:  Team  Information  Exchange 

■The  actual  snatch  and  grab  would  be  possibility  for  SOF  but  we  would  need  the  intelligence  assist. 
■ISRC:  for  your  information,  @LOCATION@  is  now  SOF  mission;  reconnaissance  a/c  #1  to  provide  over 
watch,  SOF  is  in  contact  with  a/c  now. 

TKS:  Team  Knowledge  Sharing 

■Self  defense  applies  for  hostile  acts  from  (Country  #3)  fighters  in  (Country  #  2)  or  (Country  #4) 
airspace 

■Enemy  forces  that  employ  ordnance,  Electronic  Attack  (EA),  or  force  control  systems  (achieve  a  radar 
lock)  against  friendly  forces  have  committed  a  hostile  act. 

TSOG:  Team  Solution  Option  Generation 

■Awaiting  radiological  impact  assessment  on  watershed  if  strike  building.  Second  option  in  work  is 
deny  (destroy)  local  roads  to  prevent  access  in/out. 

■If  we  crater  the  runway  and  taxiways,  we  may  be  able  to  effectively  stop  the  target. 

■To  shorten  timeline  for  tactical  tomahawk  we  can  launch  to  loiter 

Chat  log  Entries  Coded  as  Macrocognitive  Processes  in  the 

Model  of  Team  Collaboration 


TEN  A:  Team  Evaluation  and  Negotiation  of  Alternatives 

■Target  Duty  Officer  (TDO):  Just  throwing  this  out  there,  but  if  you  target  the  roadways,  is  there  a 
chance  you  could  spook  them  and  they  might  fire  off  their  missiles  and  run? 


TPPR:  Team  Process  and  Plan  Regulation 

■Not  used 

Internalized  Team  Knowledge 

TKS:  Team  Knowledge  Similarity 

■SIMISM.  C2WSPTT  always  expends  all  weapons  on  attack 

■He  wouldn't  request  return  to  base  he’d  tell  you  he  is  returning  to  base _ 

TKR:  Team  Knowledge  Resources 

■Interpositional  Knowledge  (IK):  I  remember  sketchy  authentication 

■Individual  situation  awareness  (ISA):  a/c  #2  is  out  of  position,  looks  like  other  strike  assets  quicker 

Externalized  Team  Knowledge 
ECSA:  Externalized  Cue-strategy  Association 

■The  DEC  Chief  stated  that  if  there  is  an  erect  launcher  in  a  JSOA  his  "ROE"  is  to  kill  it  ASAP  and  if  there 
is  time  to  deconflict  with  the  teams 

■He  mentioned  TLAMs  wouldn't  be  deconflicted  either,  but  I  dispute  that  logic.  First,  we  wouldn't  use 
a  TLAM  shot  to  kill  a  launcher  I  don't  think.  Unless  it  was  a  last  resort. 

■Can  get  SOF  Team  to  location  as  additional  resource  if  we  elect  to  monitor  the  site  for  any  potential 
leadership  meetings  that  may  occur  later  given  that  location  is  used  for  meetings  and  (target)  is  there 
now. 


Chat  log  Entries  Coded  as  Macrocog nitive  Processes  in  the 

Model  of  Team  Collaboration 


PRTA:  Pattern  Recognition  and  Trend  Analysis 

■Co,  TDO,  looks  like  ******  may  be  similar  to  our  first  target  with  regards  to  unknown  presence  of 
Radiological  containers  in  facility.  We  would  look  at  interdiction  for  containment  to  prevent  travel 
to/fm  that  site,  your  thoughts  on  best  plan/option 


UR:  Uncertainty  Resolution 

■TLAMs  most  definitely  have  to  be  de-conflicted  even  for  over  flight  of  the  Joint  Special  Operations 
Area  (JSOA)-unless  direct  otherwise  by  the  Joint  Force  Commander  (JFC) 


Team  Problem  Solving  Outcomes 

QOP:  Quality  of  Plan 
•Not  used 

EPP:  Efficiency  of  Planning  Process 

•Not  used 

EPE:  Efficiency  of  Plan  Execution 
•Not  used 


New  Coding  Category:  Decision  to  Take  Action 


A  new  macrocognitive  process  emerged  during  the  coding  process: 

•  Deciding  to  take  action  is  viewed  as  both  a  macrocognitive  process  and  a 
product  of  team  collaboration. 

•  Many  critical  tasks  include  taking  action  in  addition  to  developing  new 
knowledge  and  agreeing  on  a  final  solution. 

-  Various  actions  are  taken  as  part  of  information  gathering  process 
(e.g.,  MIOs,  air  warfare,  firefighters,  NORAD-FAA,  AOC,  etc.). 

-  Dynamic  decision-making  tasks  entail  a  series  of  decisions  as  part 
and  parcel  of  problem  solving. 

-  Many  tasks  involve  an  interleaving  of  knowledge  building,  decision 
making  and  taking  action  in  order  to  accomplish  the  mission. 

•  A  constant  interplay  exists  between  sharing  information  to  develop  new 
knowledge  and  maintain  situation  awareness  and  then  executing,  or 
implementing  actions,  followed  by  monitoring  and  building  new 
knowledge  on  the  unfolding  situation. 

-  Execution  of  the  mission,  would  come  to  a  screeching  halt  without 
this  continual,  iterative  cycle  of  developing  knowledge  of  the 
situation  and  responding  to  the  current  situation  by  taking  various 
actions  that  move  the  problem  along. 


